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Foreword 



T he Research and Development (R&D) series of reports at the National Center 
for Education Statistics has been initiated to 

• Share studies and research that are developmental in nature. The results of such 
studies may be revised as the work continues and additional data become 
available; 

• Share the results of studies that are, to some extent, on the cutting edge of 
methodological developments. Emerging analytical approaches and new 
computer software development often permit new and sometimes controversial 
analyses to be done. By participating in frontier research, we hope to contribute to 
the resolution of issues and improved analysis; and 

• Participate in discussions of emerging issues of interest to educational researchers, 
statisticians, and the Federal statistical community in general. 

The common theme in all three goals is that these reports present results or 
discussions that do not reach definitive conclusions at this point in time, either 
because the data are tentative, the methodology is new and developing, or the topic 
is one on which there are divergent views. Therefore, the techniques and inferences 
made from the data are tentative and subject to revision. To facilitate the process of 
closure on the issues, we invite comment, criticism, and alternatives to what we have 
done. Such responses should be directed to 

Marilyn Seastrom 
Chief Statistician 
Statistical Standards Program 
National Center for Education Statistics 
1990 K Street NW 
Washington, DC 20006-5651 




Executive Summary 



I n late January through early March of 2003, the National Assessment of 
Educational Progress (NAEP) grade 4 and 8 reading and mathematics 
assessments were administered to representative samples of students in 
approximately 100 public schools in each state. The results of these assessments were 
announced in November 2003. Each state also carried out its own reading and 
mathematics assessments in the 2002-2003 school year, most including grades 4 and 
8. This report addresses the question of whether the results published by NAEP are 
comparable to the results published by individual state testing programs. 



Objectives 



Comparisons to address the following four questions are based purely on results of 
testing and do not compare the content of NAEP and state assessments. 

• How do states’ achievement standards compare with each other and with NAEP? 

• Are NAEP and state assessment results correlated across schools? 

• Do NAEP and state assessments agree on achievement trends over time? 

• Do NAEP and state assessments agree on achievement gaps between subgroups? 

How do states' achievement standards compare with each other 
and with NAEP? 

Both NAEP and State Education Agencies have set achievement, or performance, 
standards for mathematics and have identified test score criteria for determining the 
percentages of students who meet the standards. Most states have multiple 
performance standards, and these can be categorized into a primary standard, which, 
since the passage of No Child Left Behind, is generally the standard used for reporting 
adequate yearly progress (AYP), and standards that are above or below the primary 
standard. Most states refer to their primary standard as proficient or meets the standard. 

By matching percentages of students reported to be meeting state standards in schools 
participating in NAEP with the distribution of performance of students in those 
schools on NAEP, outpoints on the NAEP scale can be identified that are equivalent 
to the scores required to meet a state’s standards. 




Objectives 



From the analyses presented in chapter 2, we find: 

• The median of the states’ primary mathematics standards, as reflected in their 
NAEP equivalents, is between the NAEP basic and proficient levels in both grades 
4 and 8. 

• The primary standards vary greatly in difficulty across states, as reflected in their 
NAEP equivalents. In fact, among states, there is more variation in placement of 
primary mathematics standards than in average NAEP performance. 

• As a corollary, states with high primary standards tend to see few students meet 
their standards, while states with low primary standards tend to see most students 
meet their standards. 

• There is no evidence that setting a higher state standard is correlated with higher 
performance on NAEP. Students in states with high primary standards score just 
about the same on NAEP as students in states with low primary standards. 

Are NAEP and state assessment results correlated across schools? 

An essential criterion for the comparison of NAEP and state assessment results in a 
state is that the two assessments agree on which schools are high achieving and 
which are not. The critical statistic for testing this criterion is the correlation 
between schools’ percentages achieving their primary standard, as measured by 
NAEP and the state assessment. Generally, a correlation of at least .7 is important for 
confidence in linkages between them. 1 Several factors other than similarity of the 
assessments depress this correlation. In 2003, correlations between NAEP and state 
assessment measures of mathematics achievement were greater than .7 in 41 of 46 
states for grade 8 and in 30 of 49 states for grade 4. 

One of these factors is a disparity between the standards: the correlation between the 
percent of students meeting a high standard on one test and a low standard on the 
other test are bound to be lower than the correlation between percents of students 
meeting standards of equal difficulty on the two tests. To be fair and unbiased, 
comparisons of percentages meeting standards on two tests must be based on 
equivalent standards for both tests. To remove the bias of different standards, NAEP 
was rescored in terms of percentages meeting the state’s standard. Nevertheless, as 
discussed in chapter 3, other factors also depressed the correlations: 

• Correlations are biased downward by schools with small enrollments, by use of 
scores for an adjacent grade rather than the same grade, and by standards set near 
the extremes of a state’s achievement distribution. 



1. A correlation of at least .7 implies that 50% or more of the variance of one variable can be 
predicted from the other variable. 



VI 



National Assessment of Educational Progress 



EXECUTIVE SUMMARY 



• Estimates of what the correlations would have been if they were all based on 
scores on non-extreme standards in the same grade in schools with 30 or more 
students per grade were greater than .7 in 42 of 43 states for grade 8 and in 37 of 
46 states for grade 4 2 

Do NAEP and state assessments agree on achievement trends over 
time? 

Comparisons are made between NAEP and state assessment mathematics 
achievement trends between 2000 and 2003. Achievement trends are measured by 
both NAEP and state assessments as gains in school-level percentages meeting the 
state’s primary standard. 3 

From the analyses presented in chapter 4, we find: 

• For mathematics achievement trends from 2000 to 2003, there are significant 
differences between NAEP and state assessment trends in 14 of 24 states in grade 
4 and 11 of 22 states in grade 8. 

• In aggregate, in grade 4 but not in grade 8, mathematics achievement gains from 
2000 to 2003 measured by NAEP are significantly larger than those measured by 
state assessments. 

• Across states, there was a positive correlation between gains measured by NAEP 
and gains measured by state assessments (r = .52 at grade 4 and r = .36 at grade 8). 

Do NAEP and state assessments agree on achievement gaps 
between subgroups? 

Comparisons are made between NAEP and state assessment measurement of 
mathematics achievement gaps in grades 4 and 8 in 2003. Comparisons are based on 
school-level percentages of Black, Hispanic, White, and economically disadvantaged 
and non-disadvantaged students achieving the state’s primary mathematics 
achievement standard in the NAEP schools in each state. 

From the analyses presented in chapter 5, we find: 

• In 34 of 70 gap comparisons at grade 4 and 17 of 62 gap comparisons at grade 8, 
NAEP found significantly larger gaps than the state assessment did. In only two 
of the comparisons (both at grade 8) did the state assessment record a 
significantly larger gap. 

• The tendency for NAEP to find larger gaps in mathematics achievement than 
state assessments did was equally strong with respect to Black- White and 
Hispanic- White gaps and slightly weaker for poverty gap comparisons. 



2. Three states for which state reports of percentages meeting standards were unavailable were not 
included in the computations of these estimates. 

3. To provide an unbiased trend comparison, NAEP was rescored in terms of the percentages meeting 
the state’s primary standard in the earliest trend year. 
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